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ABSTRACT 


Quantway is a Carnegie Math Pathways initiative which redesigns the content, pedagogy, and 
structure of traditional developmental math courses to simultaneously tackle traditional 
barriers of student success and support a broader range of developmental students in achieving 
their math potential. Specifically, Quantway is a quantitative reasoning sequence that is 
comprised of a single term developmental math course called Quantway 1 and a college-level 
math course called Quantway 2. This study assessed the effectiveness of the developmental 
math course, Quantway 1, during its first 6 semesters of implementation. We used a 
hierarchical linear modeling technique to conduct propensity score matching across 37 student 
characteristics in order to compare the course performance of Quantway 1 students with 
matched comparison students in traditional developmental math courses. Quantway 1 students 
demonstrated significantly higher odds of success than matched comparison students in 
fulfilling developmental math course requirements. Additionally, Quantway 1 effects were 
positive across all sex and race/ethnicity subgroups as well as in nearly all classrooms and 
colleges. This study provided robust evidence that Quantway 1 increases student success in 
fulfilling developmental math requirements and advances equity in student outcomes. 
Directions for future work are suggested. 
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Traditional developmental or remedial math sequences serve as a huge impediment for 
community college students, often preventing them from obtaining technical credentials and 
associate degrees, as well as blocking their transfer to four-year institutions. Nearly 60% of 
community college students nationwide are required to take at least one developmental 
mathematics course, and 80% of these students do not complete a college math course within 
three years (Bailey, Jeong, & Cho, 2010). Students spend long periods of time repeating courses 
and accruing student loan debt, ultimately leaving college without a degree. 


This crisis in completion has negative ramifications for community college student earnings and 
for American workforce development overall. Students who do not complete higher levels of 
education have significantly lower incomes. In 2012, for all people aged 25 or older, high 
school graduates earned an annual income of $33,904, those with associate’s degrees earned 
$40,820, and those with bachelor’s degrees earned $55,432 (Johnstone, 2013). Lower levels of 
educational attainment also have a detrimental effect on the overall economy as technological 
advances and global competition have created a press for an increasingly skilled workforce. The 
Georgetown Center on Education and the Workforce estimates that 65% of all American jobs 
will require a postsecondary education beyond high school by 2020, but the United States will 
fall short of meeting these requirements by 5 million workers at the current rate of production 
(Carnevale, Smith, & Strohl 2013). Community colleges play a critical role in workforce 
development, serving half of the 6.5 million undergraduates in the United States. 


Additionally, traditionally underserved students are disproportionately likely to encounter 
developmental math as a stumbling block on the road to college completion. The community 
college student population is more racially diverse, older, and lower income than 4-year 
university students (Bueschel, 2004). Minority students are placed in more developmental math 
courses and less likely to complete these courses to achieve college-level math credit than 
white students (Bailey et al., 2010; Chen, 2016). Improving the success rates of students in 
developmental math sequences is a key lever for advancing an equity agenda. 


Several aspects of traditional developmental math sequences have been proposed as 
contributors to negative student outcomes. Traditionally, students must take long multi-course 
sequences of increasing levels of difficulty to fulfill developmental math requirements. A 
sequence of multiple developmental courses, such as basic arithmetic, pre-algebra, elementary 
algebra, and then intermediate algebra, leads into a college-level transferable class such as pre- 
calculus. This structure drastically hinders student completion. Even when students complete 
one course in a sequence, many fail to enroll in subsequent courses (Cullinane & Treisman, 
2010; Bailey et al., 2010). 


Additionally, the instruction in many math classrooms does not incorporate research-based 
curriculum design and pedagogic practices that foster deeper student learning and engagement 
(Mesa, 2011). Traditional math courses emphasize transmission of content over a more 
participatory approach (Edwards, Sandoval, & McNamara, 2015), factual and procedural 
knowledge over conceptual knowledge (Mesa, 2011), and do not demonstrate the relevancy of 
mathematical concepts (Carnevale & Desrochers, 2003). Furthermore, traditional 
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developmental math courses do not address either language and literacy or non-cognitive 
barriers (e.g., math anxiety and stereotype threat) that impede many students’ ability to learn 
math (Blackwell, Trzesniewski, & Dweck, 2007; Haynes, Perry, Stupnisky, & Daniels, 2009; 
Gomez, Rodela, Lozano, & Mancevice, 2013). More recently, the relevancy of the algebra-heavy 
content of traditional math curriculum has also been called to question. A study on the Survey 
of Workplace Skills, Technology, and Management Practices found that only 19% of employees 
use any algebra in their work (Handel, 2007). 


To spur progress on this problem, Carnegie Foundation for the Advancement of Teaching 
convened a networked improvement community (NIC) - a national community of community 
college administrators and faculty, and educational researchers (Bryk, Gomez, Grunow, & 
LeMahieu, 2015). Through an improvement science approach, the NIC redesigned the content, 
pedagogy, and structure of traditional math sequences to increase the number of students 
completing their math requirements. The result of this work was two accelerated alternatives 
to traditional developmental math sequences for non-STEM students: Statway and Quantway. 
Statway is an accelerated year-long introductory college-level statistics course that integrates 
developmental math content. A previous study demonstrated Statway’s efficacy and impact on 
student success over traditional developmental math programs (Yamada & Bryk, 2016). 


Quantway provides another alternative to the traditional math sequence focusing on 
quantitative reasoning. As shown in Figure 1, Quantway 1 is a one-term quantitative reasoning 
course for students who place two levels below college-level, thus enabling them to complete 
their developmental math requirements in a single term. Students who successfully complete 
Quantway 1 are prepared for college level math and eligible to enroll in Quantway 2’ or 
another college-level quantitative reasoning course. 


Since the Quantway 1’s launch in 2012, Quantway 1’s implementation has grown more rapidly 
than Statway, with Quantway 1 enrollment growing from 418 to 1936 students over the first 
four years of implementation (Huang, Hoang, Yesilyurt, & Thorn, 2016). Because Quantway 1 
fulfills developmental math requirements in one term, community colleges can easily integrate 
it into their current developmental math offerings. Quantway 1 has been successful in 
promoting student success in developmental math courses, essentially doubling the success 
rates of traditional developmental math courses in half the time. Notably, Quantway 1 
maintained these high success rates even as student enrollment more than quadrupled. In the 
most recent 2014-2015 academic year, 57% of the students enrolled in Quantway 1 successfully 
completed the course in one term. In comparison, only 21% of a baseline group of 
developmental math students completed the traditional developmental math course in one 
year’ (Huang et al., 2016). 


* Five institutions have implemented Quantway 2, serving a total of 429 students over 3 years of implementation 
(Huang et al., 2016). For the purposes of this study, we evaluated the efficacy of Quantway 1. 


>To compute this baseline success rate, we worked with institutional researchers from six of the first Quantway 
colleges. Analyses revealed that only 20.6 percent of students were able to successfully complete their 
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Despite these promising results, it is unclear whether Quantway 1’s demonstrated impact is 
attributable to the program itself, or is due to differences in the students that select or are 
placed into Quantway 1. Building upon our descriptive results, this study pursues a more 
rigorous causal analysis of Quantway 1 through a propensity score technique, separating the 
program’s effect from potential selection bias. These results would provide a stronger 
evidentiary base on which to base decisions about adopting and scaling Quantway 1. 


DESIGN OF QUANTWAY 


Quantway 1 is designed around the premise that all students are capable of learning ambitious 
mathematics and succeeding in developmental math courses with the right supports. As 
summarized in Figure 2, Quantway 1 shares a working theory of improvement with its sister 
program Statway, aiming to increase student success through working on six key drivers: (1) 
acceleration of developmental math requirements, (2) implementation of a research-based 
instructional system, (3) socioemotional supports, (4) language and literacy supports, (5) faculty 
development, and (6) participation in a NIC (for more information on the theory of 
improvement, see Yamada & Bryk, 2016). 


The way Quantway 1 addresses these drivers differs from that of Statway in order to fill a 
different niche in community college math departments and meet the needs of specific 
students. Below, we will elaborate on the features that distinguish Quantway 1 from other 
accelerated developmental math programs. 


First, Quantway 1 accelerates students’ ability to complete developmental math, getting them 
to college level math more quickly. Quantway 1 focuses on quantitative literacy, which is 
described as “the ability to adequately use elementary mathematical tools to interpret and 
manipulate quantitative data and ideas that arise in an individual’s private, civic, and work life” 
(Gillman, 2004, p. 5). These quantitative literacy concepts are codified in a set of rigorous 
learning outcomes that were collaboratively established and vetted by a committee that 
included representatives from several mathematical professional societies.> Because Quantway 
1’s learning outcomes provide students with a strong foundation in numerical and quantitative 
reasoning concepts, it has served as preparatory quantitative reasoning course for many non- 
STEM pathways and majors and as the culminating math course for technical certificate 
programs. Some colleges offer Quantway as a pathway through college level math by 
combining Quantway 1 with Quantway 2 or another college level quantitative reasoning course. 
Quantway’s flexible design can be easily integrated into current institutional structures and 
meets a variety of community college needs. 


developmental math sequence within a full year. Additionally, 28.5 percent achieved this goal after two years, 31.6 
percent after three years, and 33.3 percent after four years. For more information, see Huang et al., 2016. 

> These mathematical societies include the National Numeracy Network, American Mathematics Association of 
Two-Year Colleges, and the Mathematical Association of America. 
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Second, Quantway 1’s instructional system is designed to ground unfamiliar math concepts in 
familiar situations through contextualization. Quantway 1’s lessons use authentic, relevant 
contexts and real data to increase student motivation to learn. Quantway 1 is organized around 
three intentional themes (citizenship, healthcare, and financial literacy) that reflect everyday 
concepts and are critically important in engaging in society. By illustrating the real world 
applications of math concepts, Quantway 1 can contribute previously unsuccessful students can 
have meaningful and positive interactions with quantitative reasoning content. 


Like Statway, the Quantway 1 instructional system is designed to foster robust and sustained 
mathematical learning, emphasizing the teaching of concepts to improve both procedural and 
conceptual understanding (Hiebert & Grouws, 2007). The Quantway 1 instructional model is 
organized around three research-based learning opportunities — productive struggle, explicit 
connections, and deliberate practice. In productive struggle, faculty engage students in 
substantive mathematical tasks that encourage students to struggle with key mathematical 
concepts and solve problems that are challenging but still within reach (Hiebert & Grouws, 
2007). By productively struggling, students can make meaning of the mathematical content for 
themselves and develop strategies for engaging with the content. Explicit connections refer to 
instruction that creates opportunities for students to make connections between mathematical 
procedures and underlying conceptual knowledge. Deliberate practice aims to improve student 
performance through a series of highly structured, increasingly sophisticated, and challenging 
tasks that deepen facility with key concepts (Edwards & Beattie, 2015). These learning 
opportunities are supported by instructional practices that facilitate student discussion and 
support collaborative learning around rich mathematical problems (Edwards & Beattie, 2015; 
Edwards et al., 2015). 


Third, Quantway 1 integrates two types of research-based student supports designed to meet 
the needs of diverse student learners — productive persistence, and language and literacy 
supports. One set of supports is designed to promote students’ ability to productively persist 
through rigorous math coursework. The socioemotional intervention, which we call Productive 
Persistence, consists of a collection of student activities and faculty actions that address the 
high-leverage non-cognitive factors that promote student tenacity and effective learning 
strategies (Edwards & Beattie, 2016). NIC members worked together with social psychologists 
to iteratively develop this package of productive persistence routines, interventions, and 
practices that work to promote growth mindset, reduce math anxiety, and increase students’ 
sense of belonging. A second set of interventions is designed to support students in successfully 
grappling with the complex language and literacy demands of mathematics, with its different 
forms of representation and elaborate grammatical forms. Quantway 1 lessons embed 
language and literacy tools to support the comprehension and organization of information in 
quantitative situations. These lessons are written to avoid literacy barriers that developmental 
math students commonly face (Gomez, Rodela, Lozano, & Mancevice, 2013; Gomez et al., 
2015). 


Fourth, because the Quantway 1 curriculum and pedagogy significantly differ from traditional 
methods of teaching, Quantway 1 faculty are invited to participate in a comprehensive 
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professional development program (Edwards et al., 2015). This Faculty Support Program 
prepares faculty to teach Quantway 1 and supports them in their first year of teaching, and 
provides ongoing opportunities for instructional improvement and professional learning. 
Through online resources, faculty mentorship, and ongoing workshops, this program prepares 
faculty to effectively implement Pathways’ collaborative instructional approach, learning 
opportunities, and productive persistence and language and literacy supports. 


Fifth, Quantway 1 faculty and administrators participate in a network improvement community 
(NIC) that provides them with a collaborative learning community to support them in teaching 
and implementing Quantway 1. The NIC social structure supports community colleges faculty 
and administrators in collectively generating and disseminating practical learning about what 
works, for whom, and under what conditions to reliably deliver efficacy at scale (Bryk et al., 
2015). 


Both Quantway 1 students and faculty report that the combination of these design elements 
creates a meaningfully different math experience from traditional developmental math courses. 
In addition, the success rates of Quantway 1 students in the first four years of implementation 
are significantly higher than institutional baselines at each of the participating colleges (Huang 
et al., 2016). However, evaluating the effectiveness of the Quantway 1 program requires 
comparing Quantway 1 student success to a reasonable counterfactual that represents how 
similar students would have performed if they had not taken Quantway 1. In this study, we 
used a propensity score matching technique (Rosenbaum & Rubin, 1983) to compare Quantway 
1 students with similar students in traditional developmental math programs in the same 
institution. We conducted the propensity score analysis within a hierarchical linear modeling 
framework (Raudenbush & Bryk, 2002) to account for the nested structure of the data with 
students within institutions in the network. We conducted a follow-up sensitivity analysis to 
examine whether the effects could be explained by other unmeasured differences between the 
two groups of students. 


We also looked at variation in performance across students, classrooms and institutions. In 
contrast to typical evaluations that report only the average impact of an intervention, this study 
assessed whether Quantway 1 is effective across the range of classrooms and institutions in the 
NIC. These investigations into variation support the program’s ability to scale with efficacy (Bryk 
et al., 2015) and can inform where improvement efforts should be targeted in order to further 
increase success rates. We also examined possible differential effects of Quantway 1 across sex 
and race/ethnicity subgroups to determine the potential of the program to promote an equity 
agenda by improving outcomes across all race/ethnicity and sex subgroups. 
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METHODS 


PARTICIPANTS 

Quantway 1 was first implemented during the spring of 2012.’ The initial cohort of students 
spanned 8 community colleges across three states (Georgia, New York, and Ohio). Throughout 
the 2014-15 academic year, Quantway 1 served 5561 students from a total of 14 colleges (see 
Appendix) across eight states (Georgia, Minnesota, New Jersey, New York, Ohio, Washington, 
West Virginia, and Wisconsin; Huang et al., 2016; Sowers & Yamada, 2015). 


DATA AND STUDY DESIGN 

Institutional researchers from participating colleges provided background data on student 
characteristics, course enrollment and performance. The analytic sample of the current study 
consisted of 4,243 Quantway 1 students from 10 colleges who enrolled in a Quantway 1 course 
between the spring of 2012 and the fall of 2014, and 83,887 potential comparison group 
students from the corresponding semesters. 


First, we identified a group of comparison traditional developmental math students with similar 
characteristics to Quantway 1 students. To obtain propensity scores, we took a hierarchical 
linear modeling (HLM) approach (Hong & Raudenbush, 2005, 2006; Raudenbush & Bryk, 2002; 
Yamada & Bryk, 2016) and constructed a two-level HLM model with a total of 37 student-level 
covariates including student background characteristics and prior course taking and success 
patterns during the two years prior to the Quantway 1 term. We selected covariates based on 
prior research findings and advice from institutional researchers in the participating colleges. 
The list includes standard student background data such as sex and race/ethnicity. It has been 
shown that these characteristics tend to differentiate students’ progress in developmental 
math sequences (Bailey et al., 2010). We also matched on students’ prior course-taking history 
and performance in the past two years. Previous research demonstrated that students’ prior 
course taking history and success patterns are a more reliable indicator of students’ educational 
and career goals than their declared program of study (Jenkins & Cho, 2012). 


Table 1 presents all of the covariates used in the propensity score matching and their 
descriptive statistics before and after propensity score matching was conducted. We found a 
substantial number of unknown records for students’ date of birth when computing students’ 
age in years. To factor these cases into the propensity model, we constructed a dummy variable 
and coded missing age as 1, otherwise 0. Also, we accounted for six cohort groups by 
formulating a set of dummy variables with Spring 2014 as a reference category. The descriptive 
data on the left panel of Table 1 shows that overall Quantway 1 students have higher 
proportions of female and Hispanic students than the non-Quantway 1 students. Quantway 1 
students had more course records in the two years before taking a Quantway 1 course, 


“One college was on a quarter system until the fall of 2012 and implemented Quantway | for the first time in the 
winter of 2012. 
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suggesting that the term they took a Quantway 1 course was less likely to be their first 
semester or year. Quantway 1 students also started their developmental course(s) earlier, and 
attempted more developmental math courses and college-level courses than non-Quantway 1 
students. 


We conducted propensity score matching separately for each cohort and college by applying a 
nearest neighbor matching algorithm (Rosenbaum & Rubin, 1985). This algorithm was 
appropriate for our study because we wanted to retain as many Quantway 1 students as 
possible and had a large pool of non-Quantway 1 students for creating matches. We attempted 
to find up to five matches per Quantway 1 student (5:1 ratio matching) to maximize the best 
matches from the non-Quantway 1 student group while still maintaining precision (Ming & 
Rosenbaum, 2000). We also specified a caliper distance of up to 0.2 to reduce the risk of bad 
nearest neighbor matches based on recommendations in the literature (Austin, 2011; 
Rosenbaum & Rubin, 1985). For propensity score matching, we used the package Matchlt (Ho, 
Imai, King, & Stuart, 2011) in R (R Core Team, 2015). 


We then estimated Quantway 1’s effectiveness by comparing success rates of Quantway 1 
students with their matched comparisons using a four-level HLM model with a binary outcome. 
Success was defined as a passing grade or a grade of C or higher’ on a Quantway 1 course for 
Quantway 1 students and a developmental math course one level below college level (or 
another course deemed equivalent to a Quantway 1 course by faculty) for the matched 
comparison students. For the latter group, we tracked course outcomes over the entire 
academic year (i.e., tracking course outcomes over the fall and spring semesters for the fall 
cohorts and the spring, summer, and fall semesters for the spring cohorts). As described earlier, 
Quantway 1 accelerates traditional developmental math sequences for students placed two 
levels below college mathematics in one semester. Similar students following the traditional 
developmental math route typically complete the developmental sequence in one and a half 
years (Bailey et al., 2010; Cullinane & Treisman, 2010). Accordingly, if comparison students in 
the fall cohorts had failed a developmental math course one level below college level in the fall 
semester but passed it the following spring semester, we counted it as success. Therefore, 
analysis was conservative, providing comparison students twice as much time to reach the 
same success benchmark as Quantway 1 students. 


In this HLM framework, matched clusters (level 1) were nested within Quantway 1 students 
(level 2), who were in turn nested within Quantway 1 faculty member classrooms (level 3) 
within their colleges (level 4). Because matched comparisons were constructed for each 
Quantway 1 student, their respective comparison students were also assigned the 
corresponding Quantway 1 faculty ID. This strategy allowed us to form each faculty member’s 
classroom as a mini-experiment in which the mean outcome of their Quantway 1 students was 
compared with that of similar students who pursued traditional math courses, in order to 


5A grade of C- or higher was used for a college that employed a +/— grading scheme. 
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estimate the variability in effect among faculty within colleges. We used HLM 7 (Raudenbush, 
Bryk, Cheong, Congdon, & du Toit, 2011) for all of the HLM analyses. 


RESULTS 


PROPENSITY SCORE MATCHING 

To obtain propensity scores, we constructed a two-level Bernoulli model and estimated its 
model parameters using maximum likelihood via penalized quasi-likelihood estimation. dj is the 
probability of student / enrolling in Quantway 1 in college /. Accordingly, nj is the log-odds of 
this incident and formally expressed as: 


Level-1 Model (Student) 


Prob(QW=1| 6) = di 


logl@i/(1 - i) = ni 
Nit = Bo; + B1(COV1))) + ... + 637(COV37;)), 


Level-2 Model (College) 


Bo = Yoo + Uol, 
Bi; = V10, 


B37) = 370, 


where QW is a dummy variable indicating whether a given student was enrolled in Quantway 1 
(coded as 1) or not (coded as 0), COV1...COV37 are the set of propensity score covariates.° We 
matched a total of 12,448 comparison students to 3,992 Quantway 1 students. Table 1 
compares the descriptive statistics on each covariate before and after matching to the 
Quantway 1 group. Table 2 documents the balance in propensity score cohort by cohort for 
each college. There were no significant differences in mean propensity scores between the 
Quantway 1 and matched comparison students in any of the cohorts for each college (see t 
values). These results provide strong evidence that comparability of the groups was achieved 
on the measured covariates. 


It may be worthwhile here to mention the matched ratios we accomplished. As described 
earlier in the Method section, we attempted to find up to five matches per Quantway 1 
student. The matched ratios in the far right column suggest that in general, we identified 4 to 5 


& We initially included two covariates of college non-STEM courses (the number of courses attempted and the 
respective success rate). However, they involved collinearity with other covariates, and accordingly, the model did 
not converge. Thus, we excluded them from the propensity model. 
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matches per Quantway 1 student. For some cohorts from Colleges 3 and 8, however, we 
identified fewer matches and needed to exclude some Quantway 1 students to maintain the 
comparability of the groups. It appears that both colleges have a relatively large population of 
students who were placed into developmental math courses and accordingly, more students at 
varying levels of developmental math. Therefore, it may be possible that certain kinds of 
students (e.g., those who failed developmental courses multiple times) were advised to take 
Quantway 1 so as to limit the number of appropriate students for matching. 


ESTIMATING QUANTWAY EFFECTS 

To estimate differences in success rates, we constructed a four-level Bernoulli model and 
estimated its model parameters using maximum likelihood via penalized quasi-likelihood 
estimation. jx represents the probability that student / matched with Quantway 1 student j 
associated with faculty member k’s class in college / successfully completed the developmental 
math sequence. Correspondingly, nix is the corresponding log-odds of this outcome and 
formally expressed as: 


Level 1 Model (Student) 

Prob(SUCCyx=1 | ix) = Dik 

log ijni/(1 - Pixi)] = Nik 

Nik! = Tojkt + Mayxi*(QWixi) , 
Level 2 Model (QW Student) 

Tlojki = Book! + Bork TERM jxi) + Borki(W12jx)) + Boski(S12jc) + Boaki(F12 jx) + 
Bosk(S13;ki) + Boek F13;x)) + Bo7Ki(F14 x1) + rojki, 

Tey = BroK1 + B11ki( TERM jx) + Faji , 


Level 3 Model (Faculty) 


Booki = Yooo! + Uookl, 


Bo1ki = Vo1ol, 
Boaki = Yo2ol, 
Bo3ki = Vo30l, 
Boaki = Voaol, 
Boski = Vosol, 
Boek = Vosol, 
Bo7ki = Yorol, 


B10ki = V1001 + U10Kl, 
Bi1k = V1101, 
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Level 4 Model (College) 


Vooo! = Soov0 + Vooo/ 
Yo1ol = 50100, 
Yo20! = 50200, 
Yo30/ = 50300, 
Voao! = Sooo, 
Vosol = S0s00, 
Voso! = So600, 
Yo70! = 50700, 
V100/ = 51000 + Vioo!, 


V1101 = 61100, 
where SUCC represents developmental math achievement (1 for successfully completed and O 


for not successfully completed), and QW is a dummy variable indicating whether the student 
was enrolled in Quantway 1 (coded as 1) or one of the matched comparisons (coded as 0). All 
the covariates at Level 2 were included as additional adjustment variables for the outcome. 
Term is a dummy variable indicating whether the outcome for matched comparison students 
was based on one semester (coded as 1) or the entire academic year (coded as 0). W12 to F14 
are dummy variables for the six cohort groups described earlier in the Method section.’ The 
results presented in Table 3 indicate that on average, Quantway 1 students demonstrated 
significantly higher odds of success, 2.05 (95% Cl [1.33, 3.18]°), in successfully completing the 
developmental math course than the matched comparison students. The corresponding 
estimated probabilities of success were 56.50% for the Quantway 1 group and 38.74% for the 
matched comparison group. 


VARIATION IN PERFORMANCE 

The estimated coefficients between the intercept and the slope at both college and faculty 
levels were negative (-.70 and -.41, respectively), suggesting that the lower the outcome for the 
matched comparison group, the larger the effect of Quantway 1. This inverse relationship was 
stronger at the college level. 


In addition, we found variation in Quantway 1 effect among colleges and faculty members (0.35 
and 0.20 for the college and faculty variances). Figures 3 and 4 display the variation in 
Quantway 1 effect size at the college and faculty levels, respectively. In both charts, we added 
three lines as references. The center line represents the average effect of Quantway 1, and the 
upper and lower lines represent the upper and lower bounds of the average effect (which are 
deviated in two SEs from the center line). Figure 3 demonstrates that there were positive 
Quantway 1 effects on student outcomes in all but College 10 (which showed no effect of 


’ We also constructed the same four-level model with individual students’ propensity scores included in the level 1 
model and those six cohort group variables added to the level 2 model for the slope. The results from this model 
revealed no significant coefficients of these additional covariates and closely mirrored those from the simpler model. 
For the ease of interpretation, we focus here on the results from the simpler model. 

* HLM 7 generates 95% confidence intervals of odds ratios. 
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Quantway 1). College 8 stands out as a positive deviant with a Quantway 1 effect outside the 
upper bound of the average effect. Figure 4 shows the variation in Quantway 1 effectiveness 
across the classrooms in the NIC. The vast majority of Quantway 1 faculty at College 8 
drastically outperformed the average Quantway 1 faculty, suggesting internal coherence at this 
institution. In contrast, a wide range of variation was observed among faculty members at 
College 3. Understanding the mechanisms that enable consistently high performance and 
investigating the causes of variation are areas of future study. 


SUBGROUP ANALYSIS 

To examine possible differential effects of Quantway 1 by sex and race/ethnicity subgroups, we 
constructed a four-level HLM similar to those described above. In this subgroup analysis, 
however, we applied effect coding to the grouping variables in order to directly represent both 
main and interaction effects on the outcome. The reference categories were female and White. 
Each of these was coded as -1. We excluded cases with the unknown sex status. Figure 5 
presents the model-based results transformed back into their natural metrics of proportion of 
students successfully completing a developmental math sequence. This metric transformation 
was made for the ease of interpretation. Positive effects of Quantway 1 were observed for each 
race/ethnicity group. Black and Hispanic male students that take Quantway 1 exhibited the 
largest increase in completion rates relative to baseline levels of performance. 


SENSITIVITY ANALYSIS 

In general, Quantway 1 effects were strong and prevalent for all subgroups. The validity of 
these effects was based on an assumption of a strongly ignorable treatment assignment. In 
other words, all relevant covariates were included in the propensity score analysis, so that the 
bias due to unmeasured covariates could be ignored. Thus, we examined the sensitivity of the 
estimated Quantway 1 effects to possible confounding by unmeasured variables (Hong & 
Raudenbush, 2005, 2006; Lin, Psaty, & Kronmal, 1998). Given some unmeasured covariates (U), 
the Quantway 1 effect (6) can be re-estimated by adjusting for hypothesized hidden bias 
(V(E[U;z]-E[Uo])) as 6 =6- y(E[Uz]-E[Uo]), where y is the unmeasured covariates’ association with 
the outcome and E[U;]-E[Uo] is their association with treatment assignment (i.e., Quantway or 
non-Quantway enrollment). 


Adapting the approach of Hong and Raudenbush (2005, 2006), we operationally defined a 
proxy for y as a coefficient derived from a four-level model designed to predict the outcome 
with the same set of covariates used in the propensity score analysis and E[U;]-E[Uo] as the 
observed mean difference between the Quantway and non-Quantway groups on the 
corresponding covariate. We then selected the largest positive value of the product of these 
two values as the largest possible bias’ and obtained an adjusted Quantway estimate (5). 


° We used the sum of the product values for those requiring a set of dummy variables (e.g., cohort group, 


race/ethnicity). 
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Accordingly, we re-estimated a Quantway 1 effect on the outcome and constructed a 95% 
confidence interval for the adjusted estimate. The adjusted estimate was .67 in logits (95% Cls 
[.26, 1.08]), and the corresponding confidence interval did not contain O or any negative value, 
thereby supporting the strong ignorability assumption. Our sensitivity analysis concludes that it 
is very unlikely that our general conclusion regarding Quantway 1’s positive effects was 
influenced by the omission of unmeasured confounding factors. 


DISCUSSION 


This study assessed Quantway 1’s effectiveness for community college students across six 
semesters of implementation using a rigorous causal analysis. A propensity score matching 
technique (Rosenbaum & Rubin, 1983) within an HLM framework (Raudenbush & Bryk, 2002) 
allowed us to control for possible selection bias by matching Quantway 1 students with 
comparable students enrolled in traditional developmental math sequences across 37 student 
characteristics. We also undertook a sensitivity analysis to examine the possibility that the 
estimated effects were influenced by unmeasured confounding variables. Throughout these 
analyses, Quantway 1 students demonstrated significantly higher odds of success than matched 
comparison students. Our analyses also indicated that these results were not due to 
unmeasured differences between the two groups. We conclude that Quantway 1 substantially 
improves student success rates in fulfilling developmental math course requirements. 


While typical evaluations may stop at estimating the effect size of Quantway 1, this study also 
sought to understand the variation of the Quantway 1 program across different colleges, 
faculty, and student subgroups. In order to achieve efficacy at scale, Quantway 1 must not only 
produce a positive effect on average, but must also be effective for diverse student populations 
across a range of different classroom and institutional contexts. Our analysis found that 
Quantway 1 effects were positive across all sex and racial/ethnic subgroups of students. As 
previously stated, students from traditionally underserved groups, including Blacks, Hispanics, 
and low-income students, are more likely to participate in developmental math courses (Chen 
2016). Because ineffective traditional developmental math courses have a disproportionate 
impact on these traditionally underserved groups, these results suggest that Quantway 1 can 
play an important role in increasing the overall number of traditionally underserved students 
completing their math requirements. In addition, Quantway 1 has a positive effect in nearly all 
classrooms and colleges in the network, indicating that the program can work for varied faculty 
in different institutional contexts. 


Quantway 1’s design includes key levers that may explain why it better supports traditionally 
underserved community college students. First, Quantway 1 is a quantitative literacy course 
that accelerates student progress to college-level math by offering developmental math 
requirements in a single term. Second, Quantway 1’s research-based instructional system 
contextualizes math concepts and is organized around three learning opportunities to promote 
rich mathematical learning for a broader range of students. Anecdotal accounts from students 
indicate that Quantway 1’s unique mathematical experiences help them see themselves as 
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mathematical learners and doers. Third, Quantway 1 aims to support socioemotional skills and 
provide language and literacy supports to help students grapple with the complex language of 
mathematics. Finally, Quantway 1’s Faculty Support Program and the NIC structure support 
faculty in implementing Quantway 1’s unique pedagogical practices across different faculty and 
institutional contexts. 


Our results suggest that Quantway 1’s comprehensive and systematic approach to tackling the 
typical barriers that developmental math students face is key to its success. Further empirical 
evidence is needed to connect particular design elements to the positive effects of the 
program. For now, we can conclude that the Quantway 1 package is an effective alternative to 
the traditional developmental math sequence and accelerates the ability of a diverse range of 
students to complete their developmental math requirements in a variety of contexts. These 
results and Quantway 1’s flexible single-term structure demonstrate its significant potential to 
positively impact numerous students in a variety of community college contexts. 


It is worth noting a couple of the key limitations of the current study that would be fruitful 
topics for future investigations. One opportunity for future research is to track college-level 
math achievement between the two matched groups one year after Quantway 1 enrollment to 
determine if Quantway 1 students perform comparably or better than matched student groups 
in future college-level math courses. Since Quantway 1 is designed to not only get students 
through their developmental math sequences but to prepare them to meet their college math 
requirements, this will be a particularly important analysis in determining Quantway 1’s 
effectiveness. As longitudinal data become available, we also plan to track longer-term 
outcomes, such as transfer rates and academic success of Quantway 1 students in 4 year 
institutions. 


The significant variation in outcomes across faculty and colleges demonstrated in Figures 3 and 
4 presents another opportunity for further investigation. The goal of quality improvement is to 
reduce the variation between classrooms and colleges achieving positive results across diverse 
contexts. Investigating positive and negative deviants provides insight into the key sources of 
variation. College 8, for example, significantly outperforms the other colleges in the network 
and maintains high performance across all the classrooms in the college. College 10, in contrast, 
performs significantly worse than other colleges. Future research should explore whether these 
colleges differ in how they enact the key design elements described above, and study the 
various adaptations that these colleges made in response to their local context. Discovering and 
sharing key practices across NIC colleges would enhance the network’s ability to replicate 
Quantway 1’s positive outcomes as it spreads to more diverse contexts. 


In conclusion, by redesigning the content, pedagogy, and structure of traditional developmental 
math courses, Quantway 1 provides a rich mathematical experience for a broader range of 
developmental students, including historically underserved groups. These efforts have 
contributed to Quantway 1’s positive impact on equitable outcomes by improving completion 
rates for all sex and racial/ethnic subgroups and across diverse contexts. Despite great 
advances in increasing developmental math completion rates on average, variation in outcomes 
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across faculty and colleges in the NIC indicates there is still much work to be done in advancing 
Quantway 1’s efficacy reliably at scale. By leveraging the NIC structure, we will continue to 
accelerate learning through quality improvement to solve this developmental math crisis. 
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APPENDIX 
List of Participating Colleges 


Atlantic Cape Community College 
Borough of Manhattan Community College 
Cuyahoga Community College 

East Georgia State College 

Madison College 

Marshall University 

Onondaga Community College 
Ridgewater College 

Rockland Community College 

Sinclair Community College 

South Georgia State College 

University of North Georgia, Gainesville 
University of Washington, Bothell 
Westchester Community College 
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Table 1. 


Descriptive Statistics of Covariates in the Two-Level Propensity Model 


Sample before matching Sample after matching 
Non-Quantway 1 Quantway 1 Non-Quantway 1 Quantway 1 
% % % % 

Sex 

Female* 56 62 60 61 

Male 44 37 39 38 

Unknown 0 1 1 1 
Race/Ethnicity 

Asian 3 4 3 4 

Black 31 30 32 30 

Hispanic 18 26 21 26 

White* 36 34 36 33 

Multiracial 1 1 1 1 

Other 1 0 0 0 

Unknown 9 5 6 5 
Any course records in past two years 

No* 45 38 46 41 

Yes 55 62 54 59 


Cohort group 
Winter 2012 6 2 3 2 
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Spring 2012 16 11 16 11 
Fall 2012 14 13 20 13 
Spring 2013 16 20 14 20 
Fall 2013 16 21 20 21 
Spring 2014* 17 15 14 15 
Fall 2014 15 19 14 19 
Age missing 21 14 23 15 
M SD M SD M SD M SD 
Age (in years) 24.98 8.47 24.09 8.18 24.09 7.55 23.97 8.11 
aia first developmental 0.92 1.70 1.62 2.46 127 2.12 137 2.25 
Course load 3.69 1.20 3.95 1.14 3.95 1.19 3.93 1.14 
Developmental math 
One level below college level 
Number of courses attempted 0.09 0.31 0.21 0.55 0.15 0.41 0.17. 0.47 
Success rate 0.00 0.04 0.00 0.04 0.00 0.04 0.00 0.03 
Two levels below college level 
Number of courses attempted 0.21 0.48 0.35 0.65 0.26 0.56 0.30 0.58 
Success rate 0.11 0.31 0.19 0.38 0.13 0.33 0.17. 0.36 
Three or more levels below college level 
Number of courses attempted 0.18 0.50 0.16 0.55 0.12 0.45 0.13 0.49 
Success rate 0.10 0.29 0.08 0.27 0.07. 0.25 0.07. 0.25 
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Developmental English 
Number of courses attempted 
Success rate 

Developmental reading 
Number of courses attempted 
Success rate 

Developmental writing 
Number of courses attempted 
Success rate 

College math 
Number of courses attempted 
Success rate 

College non-math 
Number of courses attempted 
Success rate 

College STEM 
Number of courses attempted 


Success rate 


0.10 
0.06 


0.07 
0.05 


0.10 
0.07 


0.03 
0.01 


2.10 
0.33 


0.30 
0.07 


0.37 
0.24 


0.28 
0.21 


0.32 
0.25 


0.18 
0.07 


3.57 
0.42 


1.11 
0.24 


0.08 
0.05 


0.11 
0.06 


0.11 
0.07 


0.08 
0.01 


4.32 
0.42 


0.49 
0.09 


0.33 
0.21 


0.38 
0.24 


0.34 
0.26 


0.29 
0.08 


6.01 
0.42 


1.30 
0.26 


0.08 
0.05 


0.09 
0.05 


0.09 
0.06 


0.05 
0.00 


3.29 
0.35 


0.44 
0.08 


0.35 
0.21 


0.32 
0.22 


0.32 
0.24 


0.27 
0.06 


5.17 
0.42 


1.48 
0.25 


0.08 
0.05 


0.11 
0.06 


0.10 
0.07 


0.07 
0.01 


3.76 
0.40 


0.45 
0.08 


0.32 
0.21 


0.37 
0.23 


0.34 
0.26 


0.28 
0.07 


5.42 
0.42 


1.25 
0.25 
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Note. Terms with "*" were used as reference categories (coded as 0) when formulating dummy variables. Age was computed 
in years using a date of birth and 9/1 for the fall cohorts and 3/1 for the spring cohorts (1/1 for one winter cohort group). In 
the current propensity model, we centered age around age 18. Semesters since first developmental math course takes an 
integer, such as O, 1, 2, etc., where 0 means a student took a developmental math course for the first time in the same term 
as the Quantway 1 term, 1 one semester before, 2 two semesters before, and so on. Course load refers to the number of 
courses a student took during the Quantway 1 term. Success rate was computed by dividing the number of courses 
completed with a pass in a pass/fail grading scheme, or a C or higher (C- if a +/- grading scheme is used) by the number of 
courses attempted. 


Table 2. 
Balance in Logit of the Propensity Score for non-Quantway 1 and Quantway 1 
Students 
Sample before matching Sample after matching 
Non-Quantway 1 Quantway 1 Non-Quantway 1 Quantway 1 Natchéed 
College = Cohort n M SD n M_ SD n M SD n M SD t ratio 


1 2012 Spring 585 -2.78 0.48 43 -2.72 0.37 212 -2.74 0.32 43 -2.72 0.37 -0.42 4.93 


1 2012 Fall 470 -2.41 0.31 34 -2.20 0.54 149 -2.36 0.35 31 -2.31 0.42 -0.59 4.81 
1 2013 Fall 305 -1.88 0.19 59 -1.79 0.38 249 -1.89 0.15 54 -1.88 0.16 -0.12 4.61 
1 2014 Spring 273 -2.32 0.26 29 -2.01 0.52 112 -2.23 0.26 25 -2.16 0.34 -0.94 4.48 
2 2013 Fall 690 -2.34 0.96 69 -2.29 0.80 337 -2.31 0.75 69 -2.29 0.80 -0.16 4.88 
2 2014 Spring 270 -2.93 0.62 17 -3.02 0.27 85 -3.02 0.27. 17 -3.02 0.27 -0.01 5.00 
2 2014 Fall 402 -2.86 0.41 47 -2.99 0.36 217 -2.95 0.26 46 -2.96 0.32 0.18 4.72 
3 2012 Spring 4138 -3.38 0.39 72 -3.37 0.27 360 -3.37 0.27) 72 -3.37 0.27 -0.05 5.00 
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3 2012 Fall 3234 -2.86 0.40 177 -2.74 0.42 875 -2.76 0.38 175 -2.76 0.38 -0.07 5.00 
3 2013 Spring 3745 -2.24 0.43 584 -1.93 0.75 1057 -2.11 0.53 544 -2.06 0.59 -1.66 1.94 
3 2013 Fall 2358 -2.11 0.51 408 -1.66 0.93 739 -1.89 0.64 378 -1.84 0.72 -1.28 1.96 
3 2014 Spring 3242 -2.58 0.54 290 -2.12 0.81 559 -2.20 0.72 287 -2.14 0.78 -1.08 1.95 
3 2014 Fall 4696 -2.46 0.36 402 -1.90 0.94 368 -2.08 0.69 368 -2.07 0.71 -0.16 1.00 
4 2012 Spring 594 -3.20 0.66 38 -2.78 1.27 175 -3.03 0.88 36 -2.97 0.97 -0.33 4.86 
4 2012 Fall 570 -2.88 0.47 45 -2.40 0.83 193 -2.67 0.48 42 -2.54 0.65 -1.22 4.60 
4 2013 Spring 599 -2.20 0.48 42 -1.91 0.73 188 -2.09 0.50 41 -1.96 0.68 -1.18 4.59 
4 2013 Fall 681 -2.33 0.46 69 -2.01 0.89 278 -2.28 0.48 63 -2.20 0.66 -0.84 4.41 
4 2014 Spring 617 -2.78 0.50 49 -2.43 0.81 213 -2.65 0.57 47 -2.51 0.72 -1.16 4.53 
4 2014 Fall 827 -2.66 0.38 87 -2.48 0.72 408 -2.63 0.41 85 -2.56 0.53 -1.17 4.80 
5 2013 Fall 712 «~-3.88 0.58 6 -3.60 0.76 30 -3.61 0.68 6 -3.60 0.76 -0.02 5.00 
5 2014 Spring 648 -4.58 0.58 19 -3.65 0.93 82 -3.88 0.76 19 -3.65 0.93 -0.99 4.32 
6 2012 Spring 1601 -4.72 0.45 21 -4.74 0.35 105 -4.74 0.34 21 -4.74 0.35 -0.05 5.00 
6 2013 Spring 1481 -3.75 0.55 35 -3.44 0.87 167 -3.57 0.63 34 -3.52 0.73 -0.34 4.91 
6 2013 Fall 1534 -3.82 0.49 49 -3.49 0.78 232 -3.60 0.65 48 -3.54 0.73 -0.53 4.83 
6 2014 Spring 1436 -4.20 0.45 32 -3.76 0.87 140 -3.88 0.55 29 -3.83 0.58 -0.41 4.83 
6 2014 Fall 1954 -4.16 0.35 17 -3.44 0.85 71 ~=~-3.70 0.63 16 -3.55 0.74 -0.72 4.44 
7 2012 Spring 3675 -4.79 0.91 63 -4.49 1.02 305 -4.53 0.99 62 -4.49 1.03 -0.08 4.92 
7 2012 Fall 2929 -4.68 0.58 65 -4.47 0.67 320 -4.52 0.58 65 -4.47 0.67 -0.35 4.92 
7 2013 Spring 3831 -4.05 0.49 42 -3.92 0.52 204 -3.97 0.45 42 -3.92 0.52 -0.08 4.86 
7 2013 Fall 3566 -4.21 0.44 68 -4.17 0.36 338 -4.18 0.34 68 -4.17 0.36 -0.02 4.97 
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7 2014 Spring 4097 -4.65 0.45 29 -4.29 0.80 127 -4.54 0.47 27 -4.41 0.70 -1.11 4.70 
8 2012 Winter 4693 -5.45 1.30 70 -3.90 1.55 348 -3.93 1.52 70 -3.90 1.55 -0.16 4.97 
8 2012 Spring 1955 -4.39 0.91 108 -2.32 1.83 275 -2.91 1.22 97 -2.72 1.36 -1.23 2.84 
8 2012 Fall 3420 -3.94 0.88 124 -2.14 1.69 443 -2.51 1.30 120 -2.29 1.47 -1.48 3.69 
8 2013 Spring 3615 -3.36 0.60 129 -1.93 1.36 119 -2.13 1.17 119 -2.10 1.20 -0.19 1.00 
8 2013 Fall 3230 -3.33 0.62 119 -1.87 1.46 105 -2.23 1.19 105 -2.20 1.20 -0.22 1.00 
8 2014 Spring 3088 -3.79 0.52 118 -2.49 1.23 165 -3.16 0.74 88 -3.04 0.83 -1.10 1.88 
8 2014 Fall 3796 -3.61 0.49 158 -2.55 1.37 253 -3.07 0.97 138 -2.89 1.09 -1.60 1.83 
9 2012 Spring 683 -2.86 0.56 41 -2.88 0.70 195 -2.97 0.56 40 -2.93 0.62 -0.37 4.88 
9 2012 Fall 776 -2.75 0.34 55 -2.33 1.00 226 -2.71 0.37 50 -2.58 0.54 -1.58 4.52 
9 2014 Spring 543 -2.59 0.38 30 -2.63 0.38 142 -2.59 0.33 29 -2.62 0.38 0.38 4.90 
9 2014 Fall 574 -2.49 0.29 71 -2.36 0.45 303 -2.43 0.27 68 -2.39 0.41 -0.91 4.46 
10 2012 Spring 520 -2.36 0.52 72 -2.41 0.58 343 -2.44 0.52 72 -2.41 0.58 -0.36 4.76 
10 2012 Fall 376 -2.33 0.38 48 -2.29 0.41 229 -2.32 0.37 48 -2.29 0.41 -0.43 4.77 
10 2013 Fall 243 -1.94 0.15 39 -1.87 0.16 164 -1.91 0.14 39 -1.87 0.16 -1.33 4.21 
10 2014 Spring 291 -2.03 0.40 27 -2.13 0.32 134 -2.14 0.31 27 -2.13 0.32 -0.13 4.96 
10 2014 Fall 324 -2.12 0.31 27 -1.76 0.65 109 -2.05 0.25 22 -2.04 0.26 -0.22 4.95 


27 


CARNEGIE QUANTWAY SUCCESS 


Table 3. 


Model-Based Estimation of Quantway 1 Effect on Developmental Math Achievement 


Fixed effect 
Intercept 
Term 
W12 
$12 
F12 
$13 
F13 
F14 
Quantway 1 
Term 
Random effect at level 4 (college) 
Intercept 
Quantway 1 
Random effect at level 3 (faculty) 
Intercept 


Quantway 1 


Coefficient 
-0.49 
-0.71 

0.08 
0.04 
-0.11 
-0.19 
0.11 
0.01 
0.72 
0.30 
Variance 
0.22 
0.35 
Variance 
0.02 
0.20 


SE 
0.15 
0.14 
0.14 
0.07 
0.06 
0.06 
0.06 
0.07 


p 
.005 

<.001 
585 
531 
.073 
.004 
.068 
.834 
.003 
.317 

p 
< .001 


Odds ratio 
0.61 
0.49 
1.08 
1.04 
0.89 
0.83 
1.12 
1.01 
2.05 
1.35 

Correlation 


-0.70 


Correlation 


-0.41 


CARNEGIE QUANTWAY SUCCESS 


College 
Quantway 2 or Wer 


Quantway 1 aa 
other quantitative Credit 


reasoning 


——_____________> 


Semester 1 Semester 2 Semester 3 or more 


College 
Math 


Credit 


Traditional Math Sequence 


Figure 1. Quantway 1 vs. Traditional math sequence 
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Figure 2. Six key drivers of Quantway 
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Figure 4. Variation in Quantway 1 effect among faculty members 
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Figure 5. Model-based success rates by sex and race/ethnicity 
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